User Guide-Integration Options
PVA-ISP integration options
PVA-ISP can be integrated in a product in two main ways: through a GStreamer element or directly through a C++ application using the PVA-ISP library. Both options use the same underlying ISP algorithms and PVA execution model. The complete ISP flow is:
The processing stages are backed by PVA kernels and are orchestrated by the PVA-ISP pipeline runtime. The full application flow can also expose a histogram branch after Black Level Correction and a scene-key branch from Demosaic into Global Tone Mapping. Moreover, some stages can be removed in order to reduce latency if the application supports it.
GStreamer element
The GStreamer integration exposes PVA ISP as a standard GstCUDA-based GStreamer element for multimedia applications built on GStreamer pipelines. It is suitable for use cases such as live video streaming, video recording, display pipelines, and AI inference workflows using NVIDIA DeepStream. The element receives Bayer image data, executes the configured ISP pipeline, and produces processed output for downstream GStreamer elements.
The current implementation accepts Bayer input and produces NV12 output.
The element supports standard GStreamer system-memory buffers on its sink and source pads. In particular, the sink pad accepts video/x-bayer, while the source pad can expose video/x-raw NV12. It also provides zero-copy as in GstCUDA.
GStreamer Integration Considerations
The GStreamer element supports Bayer input and NV12 output using system memory, provided by a CUDA Host memory pool, whose memory is accessible via CPU or PVA with zero copy.
For Bayer capture pipelines, this is important because many sensors produce RAW formats above 8 bits per pixel, such as 10-bit, 12-bit, 14-bit, or 16-bit Bayer. In those cases, platform-specific handling may be required to preserve the Bayer data correctly across the multimedia pipeline.
The GStreamer element is the recommended integration method when the application already uses GStreamer for capture, processing, encoding, streaming, display, or AI inference.
Applications that need direct control over graph construction, frame ownership, or ISP stage configuration may prefer the C++ library API. For additional information about building custom ISP pipelines using the library API, see Custom_Development.
C++ application or library integration
The C++ option links the product application directly against the PVA-ISP
library, currently built as libpva_isp.so. This is the natural
integration path when the application already controls capture, buffer
allocation, parameter loading, frame scheduling, or product-specific logic in
C++.
In this model, the application builds the ISP graph using the public API. The pipeline runtime owns the stage graph, negotiated frames, internal buffers, and shared PVA execution context. The application decides which stages are present, which pads are connected, which boundary frames are externally visible, and which properties are applied.
The typical lifecycle is:
isp::Pipeline pipeline{};
auto* dec = pipeline.addStage(isp::StageType::kDecompand, "Decompand");
auto* blc = pipeline.addStage(isp::StageType::kBlackLevel,
"Black Level Correction");
auto* awb = pipeline.addStage(isp::StageType::kWhiteBalance,
"White Balance");
auto* dmc = pipeline.addStage(isp::StageType::kDemosaic, "Demosaic");
auto* gtm = pipeline.addStage(isp::StageType::kGlobalToneMapping,
"Global Tone Mapping");
auto* gamma = pipeline.addStage(isp::StageType::kGammaToneMapping,
"Gamma Tone Mapping");
auto* nv12 = pipeline.addStage(isp::StageType::kNV12Conversion,
"NV12 Conversion");
pipeline.setInput("Decompand", bayer_input_spec);
pipeline.setOutput("NV12 Conversion", nv12_output_spec);
pipeline.linkMany({dec, blc, awb, dmc, gtm, gamma, nv12});
pipeline.link(dmc, "scene_key", gtm, "scene_key");
pipeline.prepare(/*exec_stats=*/true);
uint8_t* output = pipeline.process();
The library pipeline performs:
- Stage creation and ownership.
- Pad-based graph linking.
- Frame-spec negotiation between stages.
- Internal frame allocation.
- PVA program configuration.
- Ordered stage execution.
- Optional per-stage profiling statistics.
Pipeline stages
Applications select the stages needed for their graph. Joined stages combine operations that are also available as separate stages.
| Stage | Input | Output | Notes |
|---|---|---|---|
| Decompand | Companded Bayer RAW | Linearized Bayer RAW | Expands sensor values using piecewise control points and a generated VLUT. |
| Black Level Correction | Bayer RAW | Corrected Bayer RAW; optional gains side output
|
Subtracts per-site optical black levels, rescales the remaining range, applies four Bayer-site digital gains, and can estimate white-balance gains. |
| White Balance Gains | Bayer RAW | Unmodified Bayer RAW and gains
|
Estimates white-balance gains without Black Level Correction. |
| Horizontal Binning | Bayer RAW | Bayer RAW at half the input width | Optionally combines same-color samples while preserving the Bayer pattern. |
| Histogram | Bayer RAW | 256-bin histogram | Uses one Bayer 2×2 cell per observation and can apply a histogram mask. |
| Auto Exposure | Histogram | Exposure and gain controls | Computes camera controls and can optionally write them through V4L2. |
| White Balance | Bayer RAW and optional gains input
|
White-balanced Bayer RAW | Applies manual gains or gains estimated by an upstream stage. |
| Demosaic | Bayer RAW | Planar RGB; optional scene_key side output
|
Reconstructs RGB and can provide scene statistics for tone mapping. |
| Convolution | Planar RGB | Filtered planar RGB and scene_key
|
Applies a configurable 3×3 spatial filter to each color plane. |
| Color Correction | Planar RGB | Color-corrected planar RGB | Applies a configurable 3×3 color matrix. |
| Convolution + Color Correction | Planar RGB | Filtered and color-corrected planar RGB; optional scene_key
|
Fuses the spatial filter and color matrix into one stage. |
| Global Tone Mapping | Planar RGB and scene_key
|
Tone-mapped RGB8 | Uses brightness, exposure-scale limits, and temporal adaptation controls. |
| Global Tone Mapping DLUT | Planar RGB and scene_key
|
Tone-mapped RGB8 | Uses a display-luminance lookup table and adds highlight-boost and white-point-floor controls. |
| Gamma Tone Mapping | Tone-mapped RGB8 | Gamma-corrected RGB8 | Uses a generated or supplied 256-entry LUT. |
| NV12 Conversion | RGB8 | NV12 | Converts the final RGB output for display, encoding, or streaming. |
| Input Stage | Companded Bayer RAW | Corrected Bayer RAW; optional gains side output
|
Fuses Decompand and Black Level Correction, including per-site digital gains. |
| Pre RGB | Bayer RAW and optional gains input
|
Planar RGB; optional scene_key side output
|
Fuses White Balance and Demosaic. |
| Post RGB | Planar RGB and scene_key
|
NV12 | Fuses standard Global Tone Mapping, Gamma Tone Mapping, and NV12 Conversion. |
| Post RGB DLUT | Planar RGB and scene_key
|
NV12 | Fuses DLUT Global Tone Mapping, Gamma Tone Mapping, and NV12 Conversion. |
More information on the stages is available in Supported Algorithms.
